Section: New Results

Engineering of interactive systems

Participants : Caroline Appert, Michel Beaudouin-Lafon, Olivier Chapuis, James Eagan, Tony Gjerlufsen, Stéphane Huot, Wendy Mackay, Clemens Nylandsted Klokmose, Emmanuel Pietriga [correspondant] , Clément Pillias, Romain Primet.

We started working on jBricks [24] , a Java toolkit enabling the exploratory prototyping of interaction techniques and rapid development of post-WIMP applications running on cluster-driven interactive visualization platforms such as the WILD wall display (Section 8.1 ). Research on cluster-driven wall displays has mostly focused on techniques for parallel rendering of complex 3D models. There has been comparatively little research effort dedicated to other types of graphics and to the software engineering issues that arise when prototyping novel interaction techniques or developing full-featured applications for such displays. To fill this gap, jBricks integrates a high-quality 2D graphics rendering engine and a versatile input configuration module into a coherent framework, hiding low-level details from the developer. The goal of this framework is to ease the development, testing and debugging of interactive visualization applications for wall-sized displays. It also offers an environment for the rapid prototyping of novel interaction techniques and their evaluation through controlled experiments, such as the one we recently conducted about mid-air pan-and-zoom techniques for wall-sized displays (see Section 6.1 ).

We developed the Shared Substance framework for multisurface interaction [20] . It is based on Substance, which implements a novel programming model called data orientation that separates functionality from data. Shared Substance extends Substance to distributed environments. It makes distribution explicit so that the programmer can dynamically add, reconfigure and remove components at runtime. An application built with Shared Substance is a collection of processes called environments that run on different machines. Environments are discovered dynamically and can appear and disappear at any time. Each environment contains a hierarchical data structure that can be shared, in whole or in part, with other environments. Sharing can be done through replication or mounting, which entail different performance trade-offs. Shared Substance also includes the Instrumental Interaction Kit (IIK) to facilitate the development of instruments in a multisurface environment  [34] , [42] . We have used Shared Substance to develop several applications for our WILD multisurface environment: Substance Canvas manages a virtual canvas that can span multiple interactive surfaces managed by different computers, such as the tiled display, interactive table and users' laptops of the WILD room; Content can be added to the canvas from various an extensible set of sources, including live applications using Scotty (see below). Substance Grise wraps an existing application for displaying 3D brain scans into a Shared Substance environment; This allows us to run 64 copies of the application, each showing a different brain scan, and synchronize the 3D orientation of the scans using a brain prop that the user turns in his hand.

We explored the notion of user interface programming at run-time to create more malleable software [18] . Rather than creating a new user interface toolkit or supporting the customization of an interactive application from outside, we explored how well-defined hooks and a few high-level constructs could allow a programmer to modify an application “from inside”, i.e. using code that is dynamically loaded by the application at run-time. Compared with existing approaches, this supports deep customization that involve the behavior of the application, not just the surface of its user interface. The Scotty prototype implements run-time interface programming in the Mac OS X environment for any application written with the native Cocoa framework. We have used Scotty to distribute the user interface of an application over multiple devices, e.g. to move the Print button of an application onto an iPhone so the user can safely print while physically close to the printer; to replace a tool palette in an application with a toolglass; to check for the presence of attachments in an email application before sending an email; to add subtitles to a video viewer that does not have this functionality. We have also used Scotty in connection with the work on Shared Substance (see above) to teleport a live vector-based representation of a running applications to the WILD wall display. The advantage of this approach over, e.g., VNC, is that the content is properly scaled, taking advantage of the full resolution of the wall.